Skip to content

Conversation

@lilnasy
Copy link
Contributor

@lilnasy lilnasy commented Nov 19, 2025

  • "Slow but working" implementation for Linter plugins: Token-related SourceCode APIs #14829 (comment)
  • Uses the parser from @typescript-eslint/typescript-estree for tokens. The source code is fully parsed and everything but the tokens and comments is discarded.
  • Tests are directly adapted from eslint.
  • Direct commits from maintainers welcome.

Tasks

Future work

  • getFirstToken
  • getFirstTokens
  • getLastToken
  • getLastTokens
  • getTokenBefore
  • getTokenOrCommentBefore
  • getTokensBefore
  • getTokenAfter
  • getTokenOrCommentAfter
  • getTokensAfter
  • getTokensBetween
  • getFirstTokenBetween
  • getFirstTokensBetween
  • getLastTokenBetween
  • getLastTokensBetween
  • getTokenByRangeStart
  • isSpaceBetween
  • isSpaceBetweenTokens

Decisions

  • @typescript-eslint/typescript-estree peer-depends on typescript. How should we package it? Currently, typescript is being made an optional dependency.
    • overlookmotel (on discord): ideally bundled, direct runtime dependency otherwise.
  • Deprecated methods are being removed altogether in ESLint 10: getTokenOrCommentBefore, getTokenOrCommentAfter, and isSpaceBetweenTokens. These are surface level deprecations: the functionality was merged with other methods (the includeComments: true option) and plugins can migrate with a one line change. I'm guessing we are targeting fairly modern, actively developed projects. Should we expose them?
  • Program range is different between ESLint and TS-ESLint. Which one should we follow?
    • overlookmotel: they are going to both change and align soon, we should update our range calculation to match when they do.

@github-actions github-actions bot added A-linter Area - Linter A-cli Area - CLI A-linter-plugins Area - Linter JS plugins labels Nov 19, 2025
@lilnasy lilnasy changed the title Feat/linter/plugins/token methods feat(linter/plugins): Token-related SourceCode APIs (TS ESLint implementation) Nov 19, 2025
@github-actions github-actions bot added the C-enhancement Category - New feature or request label Nov 19, 2025
@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch 2 times, most recently from 3e3a5bb to dd03ef0 Compare November 19, 2025 12:35
@overlookmotel
Copy link
Member

overlookmotel commented Nov 19, 2025

Deprecated methods are being removed altogether in ESLint 10: getTokenOrCommentBefore, getTokenOrCommentAfter, and isSpaceBetweenTokens. These are surface level deprecations: the functionality was merged with other methods (the includeComments: true option) and plugins can migrate with a one line change.

Cameron and I had a bit of an argument about exactly this question!

We concluded in the end to keep all the deprecated methods, to maximize compatibility with older plugins, which may take some time to get updated (or in some cases, will never get updated). Our rationale is that most of these methods are just aliases, so no maintenance burden to keep them. The only one which is slightly different from its non-deprecated "brother" is isSpaceBetweenTokens, and that's pretty simple to implement - it just treats JSXText differently (if I remember right).

I'm guessing we are targeting fairly modern, actively developed projects.

That's true, but even actively developed projects may use old unmaintained plugins.

Note: I added the stubs in #15645. That PR also added tests which illustrate the difference in behavior between isSpaceBetween and isSpaceBetweenTokens.

Copy link
Member

@overlookmotel overlookmotel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly premature review, as it's still marked as draft, but I'm keen to get it merged so I thought I'd go through it now.

Apart from the comments below, this looks good to me. Sorry for the volume of comments - most are pretty small details.

Once we're happy with getTokens impl, I think we should merge this, and we can add more methods in separate PRs. No need to do the whole API in a single PR.

A couple of points which we can also leave to follow-ups:

  1. We should ideally lazy-load @typescript-eslint/typescript-estree package only when getTokens is first called.

  2. I assume TS-ESLint's parser also generates a ScopeManager. If it does, we may as well cache it, to avoid running scope analysis again if plugin does sourceCode.getTokens() followed by getting sourceCode.scopeManager.

@lilnasy
Copy link
Contributor Author

lilnasy commented Nov 20, 2025

  1. I assume TS-ESLint's parser also generates a ScopeManager. If it does, we may as well cache it, to avoid running scope analysis again if plugin does sourceCode.getTokens() followed by getting sourceCode.scopeManager.

@typescript-eslint/parser does. @typescript-eslint/typescript-estree does not.

The former delegates to the latter for parsing and lexing. I decided to use @typescript-eslint/typescript-estree directly to instantiate scope managers and parsers granularly.

@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch from faf3560 to de1f77e Compare November 20, 2025 12:21
graphite-app bot pushed a commit that referenced this pull request Nov 20, 2025
…pshot (#15906)

Contents of this test snapshot depends on the constructor names of scope class instances. This is a little fragile, as they can change depending on how the code is bundled (see #15861 (comment)).

At some point, we'll full minify the bundle, at which point class names will be come random gibberish like `e`, `t`, `aZ`, etc.

Avoid this problem by outputting `type` field of scope objects instead.
@overlookmotel
Copy link
Member

The former delegates to the latter for parsing and lexing. I decided to use @typescript-eslint/typescript-estree directly to instantiate scope managers and parsers granularly.

Ah ha that makes perfect sense. I had misunderstood.

@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch 6 times, most recently from e23e179 to 173e5a9 Compare November 20, 2025 20:05
@lilnasy lilnasy marked this pull request as ready for review November 20, 2025 21:03
@lilnasy lilnasy requested a review from camc314 as a code owner November 20, 2025 21:03
Copilot AI review requested due to automatic review settings November 20, 2025 21:03
@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch from 9fb3811 to ab159d9 Compare November 20, 2025 21:04
@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch from 09a2693 to 56c73d9 Compare November 21, 2025 10:22
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.

Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

version: 5.2.0
typescript:
specifier: 'catalog:'
specifier: 5.9.3
Copy link

Copilot AI Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The TypeScript version has been changed from 'catalog:' to a pinned version 5.9.3. This deviates from the project's dependency management convention of using the catalog for shared dependencies like TypeScript.

While pinning the version may provide stability for the token parsing feature, it could lead to version conflicts or maintenance overhead. Consider documenting the reason for this deviation, or using a catalog entry with a pinned version if TypeScript needs to be locked to 5.9.3 for compatibility with @typescript-eslint/typescript-estree.

Suggested change
specifier: 5.9.3
specifier: 'catalog:'

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. I don't think a change in typescript's internal parsing behavior should prevent the project from type checking using the new version.

It would be unexpected if tests start failing after a routine tooling update. It's two different concerns.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the right thing to do here is. I see your logic, but also you've chosen not to pin @typescript-eslint/typescript-estree dependency, which seems maybe inconsistent.

But anyway, it's a small point. Let's merge and ponder it later!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@camc314 Any opinion on this?

@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch 5 times, most recently from d9a4425 to 1edc327 Compare November 21, 2025 14:24
Copilot AI pushed a commit that referenced this pull request Nov 21, 2025
…pshot (#15906)

Contents of this test snapshot depends on the constructor names of scope class instances. This is a little fragile, as they can change depending on how the code is bundled (see #15861 (comment)).

At some point, we'll full minify the bundle, at which point class names will be come random gibberish like `e`, `t`, `aZ`, etc.

Avoid this problem by outputting `type` field of scope objects instead.
graphite-app bot pushed a commit that referenced this pull request Nov 21, 2025
#15946 means we can now simplify the TSDown build config.

Previously we had to have 2 separate builds for `index.ts` and `cli.ts`, to avoid `assert*` functions ending up in a shared chunk, and not getting removed by minifier. Now that problem is solved, so we can switch to a single build with 2 entry points.

TSDown does create a small extra chunk for its `__toESM` and `__commonJSMin` functions, but I think that's fine.

Annoyingly it doesn't seem to be possible to tell TSDown to generate `.d.ts` files only for the `index.js` chunk, so have to delete this pointless files in build script.

The point of all this is that `RuleTester` needs to use a ton of code from `plugins.ts`, and we wouldn't want all that code duplicated in both `index` and `plugins` chunks (especially once we're bundling TS-ESlint's parser #15861). Now that code will be shared between the two entry points in a shared chunk, not duplicated.
@lilnasy lilnasy force-pushed the feat/linter/plugins/token-methods branch from 1edc327 to 2358eee Compare November 21, 2025 17:22
@overlookmotel overlookmotel force-pushed the feat/linter/plugins/token-methods branch from 2358eee to c85261d Compare November 21, 2025 17:26
Copy link
Member

@overlookmotel overlookmotel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍾 Champagne for everyone!

I've squashed and rebased on latest main, and pushed a few commits with style nits - hope you don't mind. I haven't made any substantive changes. Will merge as soon as CI passes.

@overlookmotel overlookmotel merged commit 986cac1 into oxc-project:main Nov 21, 2025
18 checks passed
@lilnasy lilnasy deleted the feat/linter/plugins/token-methods branch November 21, 2025 17:39
graphite-app bot pushed a commit that referenced this pull request Nov 21, 2025
Follow-on after #15861. Just add a comment about a potential future perf optimization.
graphite-app bot pushed a commit that referenced this pull request Nov 21, 2025
Follow-on after #15861. Reduce variable assignments by using the final value var as `hi` in the binary search.

Also, because we initialize `sliceEnd` before the loop, don't need to check if it's `undefined` later on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-cli Area - CLI A-linter Area - Linter A-linter-plugins Area - Linter JS plugins C-enhancement Category - New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants